remove iteratorFoldM fixes #1716 #1740

kailuowang · 2017-06-23T15:10:13Z

also related https://github.com/typelevel/cats/pull/1117/files#r123765689

peterneyens · 2017-06-23T15:40:59Z

tests/src/test/scala/cats/tests/FoldableTests.scala

@@ -214,17 +209,23 @@ class FoldableTestsAdditional extends CatsSuite {
  }

  test(".foldLeftM short-circuiting optimality") {
-    // test that no more elements are evaluated than absolutely necessary
+    // test that no more than 1 elements are evaluated than absolutely necessary


Should we look if we can solve this additional evaluation with @TomasMikula's suggestion?

~~You forgot to update this comment.~~ Fixed it already 👍

codecov-io · 2017-06-23T15:45:17Z

Codecov Report

Merging #1740 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1740      +/-   ##
==========================================
- Coverage   93.99%   93.99%   -0.01%     
==========================================
  Files         253      253              
  Lines        4180     4179       -1     
  Branches      155       85      -70     
==========================================
- Hits         3929     3928       -1     
  Misses        251      251

Impacted Files	Coverage Δ
core/src/main/scala/cats/data/NonEmptyVector.scala	`100% <ø> (ø)`	⬆️
core/src/main/scala/cats/instances/map.scala	`94.44% <ø> (-0.16%)`	⬇️
core/src/main/scala/cats/data/NonEmptyList.scala	`97.36% <ø> (-0.03%)`	⬇️
core/src/main/scala/cats/instances/set.scala	`100% <ø> (ø)`	⬆️
core/src/main/scala/cats/instances/list.scala	`100% <100%> (ø)`	⬆️
core/src/main/scala/cats/Foldable.scala	`98.7% <100%> (-0.05%)`	⬇️
core/src/main/scala/cats/instances/stream.scala	`94.64% <100%> (+0.41%)`	⬆️
core/src/main/scala/cats/instances/vector.scala	`97.77% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3a969e...312ee35. Read the comment docs.

kailuowang · 2017-06-23T15:49:06Z

weird, for some reason github doesn't allow me to request @TomasMikula to review.

TomasMikula · 2017-06-23T16:56:43Z

👍

However, it's still the case that most data structures can have a more efficient implementation of foldM than the default one from Foldable.

kailuowang · 2017-06-23T18:03:14Z

@TomasMikula yes, it's nice that you also mentioned it in the comment of foldM. we'll get to it in a later PR(s).

johnynek · 2017-06-23T18:51:52Z

core/src/main/scala/cats/instances/list.scala

@@ -90,9 +90,6 @@ trait ListInstances extends cats.kernel.instances.ListInstances {

      override def filter[A](fa: List[A])(f: A => Boolean): List[A] = fa.filter(f)

-      override def foldM[G[_], A, B](fa: List[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =
-        Foldable.iteratorFoldM(fa.toIterator, z)(f)


can't we maybe do this one?

def step(in: (List[A], B])): G[Either[(List[A], B), A]] = in match { case (Nil, b) => G.pure(Right(b)) case (a :: tail, b) => G.map(f(a, b)) { bnext => Left((tail, bnext)) } } G.tailRecM((fa, z))(step)

right? If we had this, we could cover the common case for List and NonEmptyList, no?

johnynek · 2017-06-23T18:54:29Z

core/src/main/scala/cats/instances/stream.scala

@@ -118,9 +118,6 @@ trait StreamInstances extends cats.kernel.instances.StreamInstances {

      override def collect[A, B](fa: Stream[A])(f: PartialFunction[A, B]): Stream[B] = fa.collect(f)

-      override def foldM[G[_], A, B](fa: Stream[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =
-        Foldable.iteratorFoldM(fa.toIterator, z)(f)


we could copy the List approach above here:

def step(in: (Stream[A], B])): G[Either[(Stream[A], B), A]] = in match { case (Stream.empty, b) => G.pure(Right(b)) case (a #:: tail, b) => G.map(f(a, b)) { bnext => Left((tail, bnext)) } } G.tailRecM((fa, z))(step)

Patten match Stream into a #:: tail will force an evaluation of the head of the tail, so I used vanilla if

johnynek · 2017-06-23T18:57:08Z

core/src/main/scala/cats/instances/vector.scala

@@ -90,9 +90,6 @@ trait VectorInstances extends cats.kernel.instances.VectorInstances {

      override def collect[A, B](fa: Vector[A])(f: PartialFunction[A, B]): Vector[B] = fa.collect(f)

-      override def foldM[G[_], A, B](fa: Vector[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =
-        Foldable.iteratorFoldM(fa.toIterator, z)(f)


Vector.tail is so slow, we are probably better off doing fa.toList.foldM here if we do the above.

kailuowang · 2017-06-23T19:43:05Z

@johnynek thanks for helping me with these, it was too lazy of me. I've updated the implemenations.

TomasMikula · 2017-06-23T19:48:56Z

core/src/main/scala/cats/instances/stream.scala

@@ -118,8 +119,17 @@ trait StreamInstances extends cats.kernel.instances.StreamInstances {

      override def collect[A, B](fa: Stream[A])(f: PartialFunction[A, B]): Stream[B] = fa.collect(f)

-      override def foldM[G[_], A, B](fa: Stream[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =
-        Foldable.iteratorFoldM(fa.toIterator, z)(f)
+      override def foldM[G[_], A, B](fa: Stream[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] = {


This is good, but now again the default implementation of foldM in Foldable is not tested.

oops. I'll correct the test.

updated. I also added the original redundant tests back to test the stream override implementation.

Here we would profit if had some machinery like mentioned in #1684.

@TomasMikula I think we are still tested with Map and Set but I agree tests for overrides would be good.

yup. I thought about that too.

peterneyens · 2017-06-23T20:18:37Z

core/src/main/scala/cats/data/NonEmptyList.scala

@@ -431,6 +431,9 @@ private[data] sealed trait NonEmptyListInstances extends NonEmptyListInstances0
      override def fold[A](fa: NonEmptyList[A])(implicit A: Monoid[A]): A =
        fa.reduce

+      override def foldM[G[_], A, B](fa: NonEmptyList[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =
+        G.flatMap(f(z, fa.head))(bnext => Foldable[List].foldM(fa.tail, bnext)(f))
+


NonEmptyList sort of gets this implementation already through NonEmptyReducible, but I guess this saves us the split.

ah, forgot about that. A single split call probably doesn't justify the code duplication. I am going to remove this.

peterneyens · 2017-06-23T20:37:43Z

core/src/main/scala/cats/instances/stream.scala

+      override def foldM[G[_], A, B](fa: Stream[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] = {
+        def step(in: (Stream[A], B)): G[Either[(Stream[A], B), B]] = {
+          val (s, b)  = in
+          if(s.isEmpty)


Scalastyle seems to want a space between the if and the (.

johnynek · 2017-06-23T20:48:03Z

👍

kailuowang · 2017-06-26T14:11:14Z

@peterneyens I think addressed your feedback. Kindly take another look?

peterneyens

Thanks @kailuowang !

remove iteratorFoldM fixes typelevel#1716

33352a2

kailuowang added the in progress label Jun 23, 2017

kailuowang requested review from peterneyens and ceedubs June 23, 2017 15:10

peterneyens reviewed Jun 23, 2017

View reviewed changes

fix foldM to not evaluate one extra uneccessary element

ba99572

kailuowang mentioned this pull request Jun 23, 2017

foldLeftM without Free. #1117

Merged

kailuowang added this to the 1.0.0-MF milestone Jun 23, 2017

kailuowang added the Source Breaking label Jun 23, 2017

update comment

9705b47

johnynek reviewed Jun 23, 2017

View reviewed changes

more efficient implementations

e2f94dc

keep the original sequence

e955595

TomasMikula reviewed Jun 23, 2017

View reviewed changes

kailuowang added 2 commits June 23, 2017 15:49

updated NonEmptyVector as well

a8b54a9

use default foldM implemenation to test short-circuit

e2f53ae

peterneyens reviewed Jun 23, 2017

View reviewed changes

TomasMikula approved these changes Jun 23, 2017

View reviewed changes

peterneyens reviewed Jun 23, 2017

View reviewed changes

style correction / remove redundant overrides

312ee35

peterneyens approved these changes Jun 26, 2017

View reviewed changes

peterneyens merged commit 4274102 into typelevel:master Jun 26, 2017

stew removed the in progress label Jun 26, 2017

kailuowang deleted the remove-iteratorFoldM branch June 26, 2017 14:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove iteratorFoldM fixes #1716 #1740

remove iteratorFoldM fixes #1716 #1740

kailuowang commented Jun 23, 2017

peterneyens Jun 23, 2017

kailuowang Jun 23, 2017

peterneyens Jun 23, 2017 •

edited

Loading

codecov-io commented Jun 23, 2017 •

edited

Loading

kailuowang commented Jun 23, 2017

TomasMikula commented Jun 23, 2017

kailuowang commented Jun 23, 2017

johnynek Jun 23, 2017

johnynek Jun 23, 2017

kailuowang Jun 23, 2017

johnynek Jun 23, 2017

kailuowang commented Jun 23, 2017 •

edited

Loading

TomasMikula Jun 23, 2017

kailuowang Jun 23, 2017

kailuowang Jun 23, 2017 •

edited

Loading

peterneyens Jun 23, 2017 •

edited

Loading

johnynek Jun 23, 2017

kailuowang Jun 23, 2017

peterneyens Jun 23, 2017

kailuowang Jun 23, 2017

peterneyens Jun 23, 2017

johnynek commented Jun 23, 2017

kailuowang commented Jun 26, 2017 •

edited

Loading

peterneyens left a comment

remove iteratorFoldM fixes #1716 #1740

remove iteratorFoldM fixes #1716 #1740

Conversation

kailuowang commented Jun 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterneyens Jun 23, 2017 • edited Loading

Choose a reason for hiding this comment

codecov-io commented Jun 23, 2017 • edited Loading

Codecov Report

kailuowang commented Jun 23, 2017

TomasMikula commented Jun 23, 2017

kailuowang commented Jun 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kailuowang commented Jun 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kailuowang Jun 23, 2017 • edited Loading

Choose a reason for hiding this comment

peterneyens Jun 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnynek commented Jun 23, 2017

kailuowang commented Jun 26, 2017 • edited Loading

peterneyens left a comment

Choose a reason for hiding this comment

peterneyens Jun 23, 2017 •

edited

Loading

codecov-io commented Jun 23, 2017 •

edited

Loading

kailuowang commented Jun 23, 2017 •

edited

Loading

kailuowang Jun 23, 2017 •

edited

Loading

peterneyens Jun 23, 2017 •

edited

Loading

kailuowang commented Jun 26, 2017 •

edited

Loading